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The  work  begun  in  [1]  on  the  application  of  relaxation 
to  ambiguous  shape  segmentation  is  extended  to  include  curve 
linking  and  gap  filling.  A  chain  coded  input  image  is  broken 
into  segments  based  on  a  measure  of  local  curvature.  Gap 
completions  linking  pairs  of  segments  are  then  proposed  and 
represented  in  a  graph  structure.  A  second  graph,  whose  nodes 
consist  of  paths  in  the  above  graph,  is  constructed,  and  the 
nodes  of  the  second  graph  are  probabilistically  classified 
as  various  object  parts.  Relaxation  is  then  applied  to  in¬ 
crease  the  probability  of  mutually  supporting  classifications, 
and  decrease  the  probability  of  unsupported  decisions.  A 
modified  relaxation  process  using  information  about  the  size, 
spatial  position,  and  orientation  of  the  object  parts  yielded 
a  high  degree  of  disambiguation. 
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1.  Introduction 


There  exist  many  image  processing  tasks  in  which  the  ob¬ 
jects  of  interest  in  a  scene  may  touch  or  overlap  each  other 
or  noise  objects.  In  some  instances  it  is  possible  to  apply 
simple  criteria  to  correctly  segment  out  the  objects  of  interest 
in  a  single  pass.  In  other  cases,  however,  simplistic  methods 
do  not  work,  and  some  other  strategy  must  be  employed.  One 
commonly  taken  approach  is  to  perform  a  state  space  search,  in 
which  parts  of  an  object  which  have  been  identified  serve  to 
guide  the  further  segmentation  of  the  image.  This  method  tends 
to  be  both  sequential  and  order  dependent.  This  paper  describes 
a  different  approach,  in  which  a  number  of  segmentation  alter¬ 
natives  are  represented  initially  in  a  graph  structure,  and  a 
relaxation  process  is  used  to  reinforce  the  correct  decisions 
and  prune  the  poor  ones.  Specifically,  a  set  of  segmentation 
points  is  chosen  along  the  object  boundary  based  on  a  curvature 
measure  [2].  Every  piece  of  boundary  curve  which  lies  between 
two  consecutive  segmentation  points  is  represented  by  a  node 
in  a  graph.  A  gap  filling  program  [3]  then  examines  the  set 
of  curves  and  adds  to  the  graph  both  a  set  of  "imaginary"  nodes 
representing  gap  completions,  and  a  set  of  arcs  linking  those 
nodes  representing  curve  segments  which  may  be  consecutive 
along  the  boundary  of  an  object  part.  Our  problem  then  becomes 
one  of  tracing  a  path  through  the  resulting  graph  which 


corresponds  to  an  object  [4].  To  do  this,  we  trace  a  multi¬ 
tude  of  short  paths  through  the  graph,  and  submit  the  cor¬ 
responding  chain  code  to  a  classifier,  where  it  is  either 
rejected  or  assigned  a  probability  vector,  each  component  of 
which  provides  an  estimate  of  the  probability  that  the  chain 
code  is  a  particular  part  of  the  object.  Those  curves  which 
survive  classification  become  nodes  in  a  second  level  graph 
structure.  Nodes  in  this  second  level  graph  are  linked  iff 
their  corresponding  curve  segments  are  consecutive.  Relaxation 
is  then  applied  to  this  graph. 


2.  Construction  of  the  graph  structure 

This  section  describes  in  more  detail  the  manner  in  which 
the  graph  structures  mentioned  in  Section  1  are  constructed. 

It  is  helpful  in  describing  the  jrocess  to  refer  to  a  simple 
example  (Figure  1) .  The  input  to  the  program  consists  of  a 
set  of  closed  curves.  Each  curve  is  represented  as  a  sequence 
of  (x,y)  coordinate  pairs.  The  sequence  of  coordinates  is 
derived  from  a  four-neighbor  chain  code.  Specifically,  if  P.^ 
and  Pj  are  two  consecutive  points  in  such  a  sequence,  then 
the  vector  Pj-Pi  is  an  element  of  the  set  {(0,1),  (0,-1),  (1,0) 
(-1,0)}.  Suppose  the  input  to  the  program  is  the  single  closed 
curve  of  Figure  1  (minus  the  dashed  lines) .  A  measure  of  local 
curvature  is  computed  at  each  point  of  the  curve.  The  measure 
used  was  the  weighted  k-curvature  method  described  in  [2].  A 
subset  of  the  input  points  is  selected  as  a  set  of  segmentation 
points.  In  our  case,  we  performed  non-minimum  suppression  on 
the  curvature  values,  and  then  selected  as  our  segmentation 
points  those  remaining  points  whose  curvature  values  were  below 
a  preset  negative  threshold.  Thus  our  segmentation  points  tend 
to  lie  at  concave  angles.  The  heavy  dots  in  Figure  1  represent 
the  segmentation  points  for  this  example. 

Every  sequence  of  consecutive  boundary  points  whose  first 


and  last  points  are  segmentation  points,  and  which  contains 
no  other  segmentation  points,  is  called  a  segment.  A  node 


is  created  in  our  graph  structure  for  each  such  segment.  In 
Figure  1  we  have  four  segments,  labeled  A  to  D. 

A  linking  program  then  examines  this  set  of  segments,  and 
may  decide  to  link  various  pairs  of  segments.  If  the  program 
decides  to  link  segment  to  segment  S2,  there  are  two  pos¬ 
sibilities.  If  the  last  point  of  is  the  same  as  the  first 
point  of  S2,  then  an  arc  is  created  in  the  graph  from  the  node 
representing  to  the  node  representing  S2.  In  the  example, 
segments  A  and  B  would  be  linked  in  this  way.  If  there  is  a 
gap  between  the  last  point  of  and  the  first  point  of  S2, 
then  a  sequence  of  points  representing  a  Straight  line  bridging 
the  gap  is  computed.  A  new  node  S.^  is  created  representing 
this  "imaginary"  segment,  and  two  arcs  are  added  connecting 
to  and  to  S2*  In  Figure  1,  segment  A  is  linked  to 
segment  C  via  an  imaginary  segment  F. 

A  graph  representing  the  end  result  of  these  operations 
is  shown  in  Figure  2.  Of  course,  the  linking  program  requires 
a  set  of  parameters  which  determine  the  linking  criteria,  and 
the  graph  which  results  will  depend  on  these  parameters.  For 
example,  we  could  set  the  parameters  so  that  segments  which 
meet  at  large  concave  angles  are  not  linked.  In  that  case  the 
arcs  A-B,  B-C,  C-D,  and  D-A  would  not  be  present.  With  a  dif¬ 
ferent  choice  of  parameters,  we  could  have  additional  imaginary 
curves  linking  A  to  D,  for  example,  or  even  A  to  itself  via 
an  imaginary  segment  in  the  same  position  as  E. 


After  the  construction  of  this  first  level  graph,  we 
can  view  our  problem  as  one  of  finding  a  path  through  this 
graph  which  represents  the  sought  for  object.  This  could 
be  done  by  a  typical  state  space  search.  This  technique, 
however,  tends  to  be  highly  sequential  and  order  dependent. 

We  have  adopted  a  different  strategy.  The  object  is  considered 
to  consist  of  a  number  of  parts.  The  program  then  searches 
many  short  paths  through  the  graph,  concatenating  the  curve 
segments  of  the  nodes  along  each  path  and  submitting  the  result 
to  a  classifier.  (Note  that  searching  a  variety  of  short  paths 
is  a  local  process  which  could  conceivably  be  carried  out  by 
an  array  of  processors  working  in  parallel.)  Those  paths  which 
survive  classification  as  possible  object  parts  become  the 
nodes  in  a  second  level  graph.  An  arc  connects  a  pair  of  nodes 
in  graph  if  their  associated  curve  segments  are  contiguous. 

A  relaxation  process  is  then  run  on  this  graph. 


3.  Determination  of  the  initial  probabilities 


Suppose  we  have  a  sequence  of  points  (P^,P2# • • . » Pn) 
representing  a  curve  to  be  classified.  We  wish  to  assign 
to  this  curve  a  probability  vector  (N,R,T,L)  where  each  com¬ 
ponent  represents  the  probability  that  the  curve  is  a  nose, 
right  wing,  tail,  or  left  wing,  respectively.  To  accomplish 
this,  the  program  approximates  the  curve  with  a  two-piece 
polygon  with  vertices  at  P^,  P^,  P  ,  where  1  <  k  <  n  and  Pk 
is  chosen  to  minimize  a  measure  of  the  error  of  fit  between 
the  polygon  and  the  point  sequence.  Curves  whose  polygon 
approximations  fail  to  meet  fixed  limits  on  size,  angle,  and 
fit  to  the  original  curve  are  rejected.  Any  curve  not  re¬ 
jected  is  assigned  a  probability  vector  based  on  a  measure  of 
its  symmetry  (the  difference  in  length  between  the  two  sides 
of  the  polygon)  and  the  degree  of  match  between  polygon  and 
curve.  The  symmetry  is  used  in  the  same  manner  as  in  [1]  to 
split  the  probability  between  three  possibilities,  nose-tail, 
left  wing,  and  right  wing.  The  value  for  nose-tail  is  then 
split  between  the  nose  and  tail  possibilities  based  on  the  fit 
error  between  the  curve  and  its  polygonal  approximation.  In 
particular,  we  specify  an  error  limit  EQ,  at  which  the  probabi¬ 
lity  for  the  curve  being  a  nose  will  fall  to  zero.  Then,  if 
the  measured  error  is  E,  we  define: 


for  E  2  Eg ,  P(nose)  =  0,  P(tail)  = 
E 

else  P(tail)  =  =—  x  nosetail 
E0 

P(nose)=  nosetail -  P (tail) 


.  The  relaxation  process 


The  relaxation  process  operates  on  the  directed  labeled 

graph  described  above.  During  each  iteration  the  probabilities 

are  updated  according  to  the  equations  below: 

Let  N  =  the  set  of  nodes  in  the  graph. 

L  =  the  set  of  labels;  in  our  case, 

L  =  {nose,  right  wing,  tail,  left  wing}. 

An=  the  set  of  predecessors  of  node  n. 

B  =  the  set  of  successors  of  node  n. 
n 

Pm(n,j)  =  the  probability  of  label  j  on  node  n 
at  the  mth  iteration. 

Given  the  Pm(n,j)  for  all  n  €  N,  j  €  L,  we  define 

maxm(n,j)  =  max  Pm(a,i)P  (n,j)P  (b,k)a(i,j,k) 
atA 
bfcB 
i,kCL 

and  then  normalize: 


Pm+l(n'j)  " 


maxm(n» j) 

I  max  (n,. 

Ikl,  m 


if  the  denominator  is  >0 


otherwise 


A  related  relaxation  scheme,  using  the  product  of  the  probabi¬ 
lities  as  used  here,  rather  than  the  more  conventional  arith¬ 


metic  average,  is  discussed  in  [51. 


Intuitively,  given  a  node  and  a  label  on  that  node,  we 
are  considering  every  legal  path  (of  one  predecessor  and  one 
successor)  through  that  node  which  uses  the  label.  We  compute 


a  measure  of  the  "goodness"  of  each  such  path.  We  then  take 
the  maximum  result,  since  if  there  is  any  very  good  path 
using  this  label,  we  want  it  to  survive  with  high  probability, 
no  matter  how  many  poor  paths  were  investigated. 


5.  Results 


Figure  3  displays  pictorially  the  results  of  applying 
this  method  to  the  input  picture  shown  in  Figure  3a.  Remember 
that  in  classifying  each  node  a  two-piece  approximating  poly¬ 
gon  is  constructed.  Figures  3b  through  3g  display  the  polygons 
associated  with  nodes  which  have  survived  the  indicated  number 
of  iterations.  After  fourteen  iterations,  the  original  graph, 
which  consisted  of  53  nodes,  is  pruned  to  17  nodes.  The  pruned 
graph  is  illustrated  in  Figure  4. 

The  number  on  each  node  is  its  number  in  the  original  graph 
of  53  nodes.  The  letter  is  the  remaining  label  assigned  to  the 
node.  Note  that  the  final  graph  consists  of  four  disconnected 
subgraphs.  Three  of  these  correspond  to  the  three  airplanes, 
and  one  is  a  noise  object.  The  graphs  for  the  second  and  third 
planes  consist  of  exactly  four  nodes  each,  one  for  each  airplane 
sub-part.  The  graph  corresponding  to  the  first  plane  contains 
two  tail  nodes.  This  type  of  ambiguity  can  arise  because  there 
may  be  several  paths  between  a  given  pair  of  nodes  in  the  first 
level  graph  and  several  of  these  paths  may  pass  classification 
as  possible  features  (Figure  5)  . 

The  noise  object  survives  because  the  relaxation  process 
as  defined  above  does  not  take  into  account  the  relative  position, 
size,  orientation,  etc.  of  the  parts.  A  triple  of  labels  has 
compatibility  1  if  it  is  a  legal  sequence,  and  0  otherwise. 


We  can  extend  the  notion  of  compatibility,  however,  to  take 
account  of  the  relationships  between  various  object  parts. 


As  before,  we  have  a  set  of  labels  L  =  ,  and 

with  each  node  we  associate  a  probability  vector  (P1,P2, . . . ,Pk) , 
where  P^  represents  the  probability  that  label  l i  is  the  cor¬ 
rect  label  for  that  node.  In  addition,  we  can  associate  with 
each  label  l i  a  .measurement  function 

ri 

f^  :  image  curve  R 

which,  given  a  curve  as  argument,  produces  an  r^  component 
measurement  vector.  Then,  at  each  node,  we  store  not  only  the 
probability  vector,  but  the  n  measurement  vectors  derived  by 
applying  each  of  the  measurement  functions  f^,...,f  to  the 
curve  segment  associated  with  that  node.  For  each  sequence  of 
labels,  l.l.Z.,  we  define  a  compatibility  function 

X  J  K 

r .  r .  r. 
a  :R1xR-ixRK-*-R 

ijk 


whose  argument  is  a  triple  of  measurement  vectors.  (if  ijk 
is  not  a  legal  sequence  of  labels,  the  function  is  identically 
0.)  If  we  let  M*  be  the  measurement  vector  obtained  by  apply¬ 
ing  measurement  function  f^  to  the  curve  associated  with  node 
n,  we  can  rewrite  eq.  1  as: 


max  (n,£ . ) 
m  j 


max 


m 


m 


m 


'k  ijk 


n 


l*i,k*  L 


For  example,  suppose  l ^  =  "left  wing" 

=  "nose" 

l k  =  "right  wing" 

Then  the  measurement  vector  M1  might  contain  measures  of  the 

cl 

size,  angle  of  sweep,  etc.  of  the  possible  left  wing  defined 

l 

by  the  curve  segment  associated  with  node  a.  Similarly,  M 
could  contain  measurements  of  the  size,  location,  orientation, 
etc.  of  the  possible  nose  at  node  n.  The  compatibility  function 
a. ..  could  then  take  into  account  such  factors  as  the  relative 

ljk 

sizes  of  the  wings,  the  similarity  between  the  angle  each  makes 
with  the  nose,  etc.  Note  that  whereas  the  value  of  the  pre¬ 
viously  define  a(i,j,k)  depended  only  on  the  label  sequence, 
the  value  of  the  a. ..  defined  above  also  depends  on  the  proper- 
ties  of  the  curve  associated  with  the  nodes  in  the  graph  where 
it  is  applied.  Because  the  value  of  a  depends  on  its  location 
in  the  graph,  it  is  said  to  be  a  "space  variant"  compatibility. 

We  might  refer  to  a  as  statically  space  variant,  in  that  although 
its  value  depends  on  its  location  in  the  graph,  the  value  of 
a,  given  a  fixed  triple  of  nodes,  remains  constant  throughout 
the  relaxation  iterations. 

The  program  was  modified  to  incorporate  the  space  variant 
compatibility  function  defined  above.  The  measurement  func¬ 
tions  and  compatibility  functions  were  particularly  simple, 
yet  yielded  a  dramatic  improvement  in  performance.  Specifically, 
the  same  measurement  function  was  used  for  all  four  labels. 


Given  a  curve  segment,  recall  that  we  formed  a  two  piece 
polygonal  approximation  in  the  process  of  classification. 

This  polygon  was  reduced  to  a  vector  as  shown  in  Figure  8 . 

The  4-tuple  representing  this  vector  is  the  measurement  vec¬ 
tor  given  by  applying  the  measurement  function.  Although 
this  measurement  represents  very  little  data,  it  is  suffi¬ 
cient  to  convey  the  general  location,  size,  and  orientation 
of  the  original  curve  segment.  We  then  defined  four  compati¬ 
bility  functions  for  the  four  legal  sequences  of  labels.  (For 
an  illegal  sequence,  a  =0).  This  function  also  was  very  simple. 
In  general,  we  would  make  the  range  of  a  be  10,1].  However, 
in  this  case,  we  let  the  range  of  a  be  just  the  set  {0,1}. 

For  a  legal  label  sequence  “ij*  would  be  1  if  a  set 

of  criteria  comparing  the  relative  sizes  and  orientations  of 
the  measurement  vectors  were  satisfied,  and  0  otherwise.  The 
result  of  applying  the  modified  method  to  the  same  input  image 
as  before  is  shown  in  Figure  6.  The  initial  graph  structure 
consisted  of  the  same  53  nodes  as  before.  Now,  however,  the 
process  stabilized  after  two  iterations  instead  of  14,  and 
arrived  at  a  superior  result.  The  final  graph  is  displayed  in 
Figure  7.  Plane  1  no  longer  has  an  ambiguous  tail.  Furthermore, 
the  noise  object  has  been  eliminated. 
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